Value iteration and optimization of multiclass queueing networks

This paper considers in parallel the scheduling problem for multi class queueing networks and optimization of Markov decision processes It is shown that the value iteration algorithm may perform poorly when the algo rithm is not initialized properly The most typical case where the initial value function is taken to be zero may be a particularly bad choice In contrast if the value iteration algorithm is initializedwith a stochastic Lyapunov function then the following hold i A stochastic Lyapunov function exists for each intermediate policy and hence each policy is regular a strong stability condition ii Intermediate costs converge to the optimal cost iii Any limiting policy is average cost optimal It is argued that a natural choice for the initial value function is the value function for the associated deterministic control problem based upon a uid model or the approximate solution to Poisson s equation obtained from the LP of Kumar and Meyn Numerical studies show that either choice may lead to fast convergence to an optimal policy Mathematics Subject Classi cation Primary B M B C Secondary E J